Arrangement and method for processing image data

ABSTRACT

A source image (S 1 ) distorted by a camera lens can be transformed into a rectified target image (T), by means of a tabular imaging specification. The above occurs during read-out from the image sensor and in real-time. Each source pixel in the source image is assigned none, one or several target pixels in the target image (T 1 ). A first controller (C 1 ) controls the image sensors (B 1 , B 2 ) accurately with tine and the image equalization and image correlation. A second controller (C 2 ) controls the first controller (C 1 ) and works in a manner temporally decoupled from the above.

CLAIM FOR PRIORITY

This is a national stage application of International Application No.PCT/DE01/02015, which was published in the German language on Nov. 28,2002 and which was filed in the German language on May 25, 2001.

TECHNICAL FIELD OF THE INVENTION

The invention relates to an arrangement and method for processing imagedata, particularly in imaging systems for vehicle occupant protectionsystems.

BACKGROUND OF THE INVENTION

Microsoft Research Technical Report MSR-TR-98-71 “A Flexible NewTechnique for Camera Calibration” discloses a method for compensatingimage distortions whereby a mathematical rule is used to map a sourceimage recorded by a camera onto a target image. The computational rulecalculates the corrected target image from the source image loaded intomain memory.

In occupant protection systems, exacting requirements are placed on thespeed of optical image recognition, as the position of a person on avehicle seat must be rapidly established in the event of an accident inorder to deploy the restraint system accordingly. The image sensorsprovided in the camera of an imaging system record images of an imagearea in close succession, the resulting image data of an image having tobe read out of the image sensor by a control unit before the next imageis recorded.

This requires a large amount of memory for storing the image data of animage, precise timing for transmitting the image data to the controlunit and considerable computing power for further processing of theimages.

SUMMARY OF THE INVENTION

An object of the invention is therefore to cost-effectively reduce theprocessing overhead for image data.

According to an aspect of the invention, a method for compensating imagedistortions is provided which can be used particularly in imagingsystems for occupant protection systems. An image distorted by theoptics of a camera system produces, in the camera system's image sensor,a source image which is distorted in different ways depending on thequality of the optics, the focal length of the camera system and otheroptical parameters. The source image is preferably broken down intoindividual source pixels each disposed at a predefined position in thesource image and whose grayscale values recorded by the image sensor arein each case stored under a predefined source pixel address in the imagesensor.

The source image is preferably mapped into a target image via apredefined mapping rule, whereby a corrected target image is produced.The target image preferably includes target pixels whose grayscalevalues are stored in each case under a target pixel address in a targetmemory, a source pixel being mapped into no target pixel or into one ormore target pixels, the grayscale value of the source pixel addressbeing transferred to the target pixel address.

The mapping rule for correcting a source image to produce a target imageis preferably stored in tabular form in a rectification table in amemory of a first control unit. The first control unit also takes overthe complex and time-precise control of the image sensor(s), therebyadvantageously enabling the mapping rule to be quickly processed. Inaddition, it is unnecessary to buffer the source image, thereby savingconsiderable memory space.

This advantageously reduces the required memory space and simultaneouslyenables correction of the source image to be performed without delay,which is particularly necessary for occupant protection systems.

Mapping of the source image into the target image according to thespecified mapping rule produces a target image having fewer pixels thanthe source image. There are therefore a number of source pixels whichare not mapped into the target pixel. In addition, the image sensorgenerally captures more information than is actually required. Thisredundant information is filtered out by the mapping rule. Filtering anddata reduction are therefore advantageously performed. Only the targetimage generated by the mapping rule is stored in the first control unit,which means that memory space is in turn saved in the first controlunit.

Two image sensors are preferably connected to the first control unit.The corrected target images are preferably correlated row-wise with oneanother in the first control unit to produce a range image containingnot only grayscale value information but also range information of therelevant image points of the camera. From the image data of the twotarget images, only part of the range image is preferably formed,buffered, and fed out cyclically or when requested by a second controlunit, thereby advantageously saving memory space.

The first control unit controls the image sensor(s), provides timingmatched to the image sensor and basically performs all the time-criticaland compute-intensive image processing operations. The resulting imagedata is fed out to a second control unit via a specified interface,preferably a standard interface such as PCI, local bus etc. The secondcontrol unit takes over the computation results of the first controlunit, e.g. the range image or parts of the range image, and controls thefirst control unit. In the second control unit, the received image datais analyzed using seat occupancy classification algorithms. It ispossible to detect, for example, the position of an occupant on avehicle seat, the position of the occupant's head, the position of achild seat or an unoccupied vehicle seat. The resulting data isforwarded to an airbag control unit (ACU).

The camera optics of an image sensor are subject to manufacturingtolerances. To compensate for the manufacturing tolerances, therectification table associated with the given camera optics ispreferably determined at the end of the production line by buffering theimage data of a reference image acquired by one of the image sensors ina first memory. This first memory can be in the first or the secondcontrol unit. Using an initialization routine, the appropriaterectification table is created and stored in this first memory so thatthe storage space of the first memory is advantageously used twice. Thisinitialization routine is preferably executed in the second controlunit. The rectification table is stored in a read-only memory of thefirst control unit. Alternatively, the rectification table is stored inthe read-only memory of the second control unit, e.g. a flash memory,and transferred to the first control unit at startup.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the interior of a vehicle with an optical imaging system;

FIG. 2 is a block diagram of an arrangement for image processing;

FIG. 3 is a flowchart of an initialization routine for compensating foroptical system tolerances; and

FIG. 4 is a flowchart of an image processing routine.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 schematically illustrates a vehicle 1 in which there ispreferably located a vehicle seat 2 having a seat pad 23, a backrest 21and a head restraint 22 mounted thereon. In the lining of the vehicleroof 3 there is disposed, preferably between the two front seats, anoptical camera system 7, 71, B1, B2, C1,C2 with which a predefined imagearea Bi of the vehicle interior can be captured. Preferably two imagesensors B1, B2 cover the image area Bi comprising the vehicle seat 2with any subject 9 located thereon via a camera optical system. In FIG.1, the subject 9 is schematically illustrated as a vehicle occupant.

In further embodiments the subject 9 can be a child seat, objects orsimilar, or the vehicle seat 2 can be unoccupied.

In the front part of the vehicle 1, under the windshield 4, there isdisposed a dashboard 5 below which there is a footwell 8 for the feetand legs of the occupant 9 and in which an airbag 26 is located. Thelower extremity of the footwell 8 is delimited by the vehicle floor 6 onwhich seat rails 24 are disposed. In the area of the lower part of theseat pad 23, the vehicle seat 2 is connected to the seat rail 24 viasupports. The vehicle seat 2 is therefore displaceably disposed in theX-direction, i.e. the vehicle direction.

The camera system 7 preferably comprises two image sensors B1, B2, alight source 71 preferably equipped with a plurality of light-emittingdiodes or at least one laser diode, and an analysis unit C1,C2. Theimage area Bi is illuminated both by the light source 71 and by anyavailable ambient light. The optical axes of the two image sensors B1,B2 have a predefined spacing L. This enables range information of thesubjects in the predefined image area Bi to the camera system 7 to beacquired from the images captured by the two image sensors B1, B2 usingstereo image processing methods. The camera 7 preferably incorporatesthe two image sensors B1, B2 and the light source 71 in a compacthousing. The analysis unit C1, C2 is likewise preferably disposed in thesame compact housing, as the volume of data transmitted by the imagesensors B1,B2 to the analysis unit C1,C2 is high. The exemplary imagesensor B1 preferably has a matrix-shaped pixel arrangement with aresolution of 320×288 pixels and a grayscale depth or grayscaleresolution of 8 bits=256 grayscale values per pixel. Using two imagesensors B1 and B2 and a minimum sampling rate of 50 images per secondper image sensor results in an overall data transmission rate betweenthe image sensors B1,B2 and the analysis unit C1,C2 of320×288×8×2×50=73.728 Mbit/s.

In another embodiment, only one image sensor B1 or B2 is provided,thereby reducing the costs. Here, the required range information ispreferably obtained from optical delay measurements or other imageprocessing methods.

FIG. 2 shows the block diagram of an image processing arrangement. Twoimage sensors B1 (left) and B2 (right) each capture an image area Bi viaan optical system OPT 1, OPT 2. As essentially identical processes occurin the two image sensors B1, B2, the image processing operation will nowbe described using the example of the left image sensor B1.

The image to be captured of the image area Bi is distorted by theoptical system OPT 1 with the result that a distorted source image S1 isproduced in the image sensor B1. The image sensor B1 is preferablycontrolled by a first control unit C1. A sensor timing unit T1 in thefirst control unit C1 supplies the necessary control signals preciselytimed for the image sensor B1. The source image S1 captured by the imagesensor B1 must be read out within a short time, e.g. at a sampling rateof 50 images per second in a few milliseconds. In addition, because ofthe analog design of the image sensor B1, the storage time of a sourceimage S1 in the image sensor B1 is short.

The image data present in the image sensor B1 is transmitted pixel bypixel to the first control unit C1, a pixel at a predefined pixeladdress containing a grayscale value. The image data supplied by theimage sensor B1 is processed by a rectification controller C13 in thefirst control unit C1. The rectification controller C13 controls thecorrection of the source image S1 to produce a target image T1. Thesource image S1 is essentially mapped into a target image T1 pixel bypixel using a rectification table TA stored in a memory M10. Thecorrected (rectified) left target image T1 and the corresponding righttarget image T2 are stored in a buffer (target memory) M11, M12 in thefirst control unit C1. A census transformer C11 reads out at least partsof the two target images T1, T2, processes them and correlates the partsof the left and the right target image T1, T2 with one another to obtainrange information of the captured image.

The correlation is preferably performed in a correlator C12 to which 6preprocessed rows of the left target image T1 and 6 preprocessed rows ofthe right target image T2 are fed. The range image AB which has beencorrelated and provided with range information is stored in a memory M0.Preferably only a few rows of the correlated image are stored ortransformed. A central control unit C10 located in the first controlunit C1 controls all the functional blocks T1, C13, MUX1, MUX2, C11, C12contained in the first control unit C1, and the memories M10, M11, M12,M0. Upstream of the target memories M11, M12 and the memory M10 thereare provided multiplexers MUX2, MUX1 with which the central control unitC10 controls the memory accesses to the individual memory areas.

The central control unit C10 is preferably controlled by a secondcontrol unit C2. The second control unit C2 is largely exempt from thetime-critical requirements for reading out the image sensors B1, B2 andsubsequent rectification and correlation of the image data and istherefore time-decoupled. Consequently, the control unit C2 can reactflexibly to external events initiated e.g. by an airbag control unit C3connected via an interface. The second control unit C2 is equipped witha main memory M2 and a nonvolatile memory M3. At the request of thesecond control unit C2, the corrected and correlated image data storedin memory M0 of the first control unit C1 is preferably transferred tosaid second control unit. In addition, the second control unit C2supplies the system clock and transmits commands (Execute) to thecentral control unit C10 of the first control unit C1. The image datatransferred by the memory M0 is further processed in the second controlunit C2. In the second control unit C2, a pattern recognition algorithmis executed by which the occupancy state of a vehicle seat is classifiedfrom the image data.

Advantageously, because of the memory M10, M11, M12 present in the firstcontrol unit C1, no external memory with a corresponding number ofrequired lines is necessary.

FIG. 3 shows the flowchart for initializing an image processingarrangement. The optical systems OPT1 and OPT2 are to be manufactured asinexpensively as possible, resulting in high manufacturing tolerances.As a result, each optical system OPT1, OPT2 is subject to differentdistortions. Using the initialization routine described below, arectification table TA pertaining to the relevant optical system iscreated for each optical system at the end of the production line. As aresult it is advantageously possible to compensate for even highmanufacturing tolerances of an optical system type series.

At the start of the initialization routine, a reference image RB is heldin a predefined position in front of the optical system OPT1 of theimage sensor B1. The reference image RB exhibits a predefined pattern,e.g. vertical and horizontal lines L2,L1 and/or dots P each occupying apredefined position. The image sensor B1 now captures the referenceimage RB, thereby producing a distorted reference image, e.g. the sourceimage S1 in the image sensor B1. The image data assigned to the sourceimage S1 is read out by the first control unit C1 and stored in thememory M10. Using a predefined computational rule, the first controlunit C1 determines the rectification table TA from the image data andstores it in the memory M10 or in the read-only memory M13 of the secondcontrol unit C2. The tabular data of the rectification table TA issubsequently copied to the memory M10 at initialization, e.g. when theoccupant protection system is activated.

In a further embodiment, the computational rule to determine therectification table TA is executed in the second control unit C2. Thisis possible, as the creation of the rectification table TA takes placeat the end of the production line and is therefore not time-critical.

The rectification table TA is now available in a read-only memory M3.Initialization is therefore complete.

FIG. 4 shows the flowchart of an image processing routine. At the startof the routine the rectification table TA is loaded from the read-onlymemory M3 of the second control unit C2 into the memory M10 of the firstcontrol unit C1, the rectification table TA being exemplary for theprocessing of a source image S1 of the image sensor B1. A rectificationtable is preferably provided for each image sensor.

The first control unit C1 reads the image data of the distorted sourceimage S1 out of the left image sensor B1 pixel by pixel. Using themapping rule stored in the rectification table TA, the data is mappedpixel by pixel into a corrected target image T1 in the rectificationcontroller C13 of the first control unit C1. The corrected target imageT1 is stored in the memory M1. The image data of the distorted sourceimage S2 is processed correspondingly. The resulting target image T2 isstored in the target memory M12.

The image data of the target images T1, T2 are preferably read outrow-wise from the memories M11, M12 and processed using a predefinedcensus transform to produce left and right census rows, preferably sixfor each left and right image, which are buffered and correlatedrow-wise with one another. The image data of the pixels of thecorrelated rows additionally contains range information and is stored inthe memory M0. On request, this image data is transferred to the secondcontrol unit C2 which now classifies the transferred image data usingpattern recognition algorithms. The classification result is transmittedto the airbag control unit C3 (ACU).

The first control unit C1 is preferably implemented as an ASIC(Application Specific Integrated Circuit) or FPGA (Field ProgrammableGate Array). The second control unit C2 is preferably implemented as amicrocontroller or microprocessor. The first and the second control unitC1, C2 can be incorporated in one housing and interconnected viaconductive tracks. In a further embodiment, the first and the secondcontrol unit C1, C2 can be integrated in a package or even on a chip. Inthe second control unit C2, triggering decisions for occupant protectionsystems can be additionally implemented.

In the first control unit C1, a large number of operations are performedin parallel, whereas in the second control unit C2 only a small numberof operations or a single operation are processed in parallel.

1. An arrangement for processing image data in imaging systems foroccupant protection systems, comprising: a first control unit which, inreal time, reads out from at least one image sensor, grayscale values ofthe image data as source image pixels; an optical system which distortsthe image data, where the first control unit corrects the image datadistorted by the optical system using a tabular mapping rule so that atarget image is produced and provides range information of the targetimage; and a second control unit for controlling the first control unitand taking over the target image data and the range information of thetarget image processed by the first control unit, the second controlunit performing seat occupancy classification, wherein by mapping thesource image into the target image according to the mapping rule, thetarget image having fewer pixels than the source image can be obtained.2. The arrangement according to claim 1, wherein the first control unitcontains a rectification table stored in a first memory and which can beused to execute the mapping rule for correcting the image supplied bythe image sensor.
 3. The arrangement according to claim 2, wherein toinitialize the rectification table, image data of a reference imagecaptured by one or more image sensors is stored in the first memory. 4.The arrangement according to claim 1, wherein the first control unit isimplemented as an ASIC or FPGA, the second control unit is implementedas a microcontroller or microprocessor, and the first and the secondcontrol units are disposed separately or one of in an IC package andon achip.
 5. The arrangement according to claim 1, wherein two image sensorsare present, the first control unit correlating the corrected image datain each case to produce a range image.
 6. The arrangement according toclaim 1, wherein at least one image sensor is linked via the firstcontrol unit to the second control unit.
 7. A method for processingimage data in imaging systems for occupant protection systems,comprising: reading the image data available in an image sensor of asource image distorted by an optical system from the image sensor by afirst control unit in which the image data is corrected in real timeusing a tabular mapping rule so that a target image is produced and isprovided with range information, whereby a seat occupancy classificationis carried out using in a second control unit which controls the firstcontrol unit, the corrected target image data and range informationtaken from the first control unit is provided with range information andclassified, wherein the target image has fewer pixels than the sourceimage.
 8. The method according to claim 7, wherein a pixel under apredefined pixel address contains a grayscale value.
 9. The methodaccording to claim 7, wherein the target image or the image data of thetarget image is buffered in a memory of the first control unit.
 10. Themethod according to claim 7, wherein if a first and a second imagesensor are present, the two corrected target images produced arecorrelated row-wise with one another to determine range information, theresulting image pixels or image data of the image pixels provided withdisposed in at least one image row, are buffered in a memory of thefirst control unit, and at the request of a second control unit, thebuffered image pixels or image data of the image pixels are transferredto the second control unit for further processing, vehicle seatoccupancy classification being performed in the second control unit. 11.The method according to claim 10, wherein the result of the vehicle seatoccupancy classification is forwarded to a central occupant protectioncontrol unit.
 12. The method according to claim 10, wherein a predefinedreference image is captured by at least one image sensor to initializeor create a rectification table, the captured reference source image isread out of the image sensor and stored in a first memory, therectification table is created from the stored reference source image,and the created rectification table is stored in the first memory,whereby the reference source image stored in the first memory is atleast partially overwritten by the rectification table.
 13. The methodaccording to claim 12, wherein the reference source image has apredefined pattern.
 14. The method according to claim 12, wherein thereference source image is captured by the two image sensors and used asreference for the correlation.
 15. The method according to claim 12,wherein a rectification table is provided for each image sensor or onerectification table is available for all the image sensors.
 16. Themethod according to claim 7, wherein a system clock of the secondcontrol unit is essentially independent of a system clock of the firstcontrol unit.
 17. The method according to claim 7, wherein imageprocessing in the first control unit is executed in parallel withclassification in the second control unit.
 18. The method according toclaim 7, wherein the first and the second control unit operate largelyindependently of one another.